Data processing systems, devices, and methods for content analysis

ABSTRACT

Systems, devices and methods operative for identifying a reference within a figure and an identifier in a text associated with the figure, the reference referring to an element depicted in the figure, the reference corresponding to the identifier, the identifier identifying the element in the text, placing the identifier on the figure at a distance from the reference, the identifier visually associated with the reference upon the placing, the placing of the identifier on the figure is irrespective of the distance between the identifier and the reference.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This patent application is a continuation of U.S. Utility patentapplication Ser. No. 16/423,027, filed on May 26, 2019, which is acontinuation of U.S. Utility patent application Ser. No. 15/047,814,filed on Feb. 19, 2016, which is a continuation of U.S. Utility patentapplication Ser. No. 14/979,395, filed on Dec. 27, 2015, which is acontinuation of U.S. Utility patent application Ser. No. 13/623,251,filed on Sep. 20, 2012, which claims a benefit of priority to U.S.Provisional Patent Application 61/633,523, filed on Feb. 13, 2012 and abenefit of priority to U.S. Provisional Patent Application 61/537,314,filed on Sep. 21, 2011.

All of the above-identified applications are herein fully incorporatedby reference for all purposes.

TECHNICAL FIELD

The present disclosure relates to systems, devices and methods for dataprocessing. More particularly, the present disclosure relates tosystems, devices and methods for aiding users in content analysis.

BACKGROUND

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present techniques,which are described and/or claimed below. This discussion is believed tobe helpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentdisclosure. Accordingly, the discussion should be understood that thesestatements are to be read in this light, and not as admissions of priorart. Likewise, in the present disclosure, where a document, act or itemof knowledge is referred to or discussed, this reference or discussionis not an admission that the document, act or item of knowledge or anycombination thereof was at the priority date, publicly available, knownto the public, part of common general knowledge or otherwise constitutesprior art under the applicable statutory provisions; or is known to berelevant to an attempt to solve any problem with which the presentdisclosure is concerned.

U.S. Pat. No. 5,774,833 is herein incorporated by reference in itsentirety.

U.S. Pat. No. 5,845,288 is herein incorporated by reference in itsentirety.

U.S. Pat. No. 8,160,306 is herein incorporated by reference in itsentirety.

A typical figure, such as an anatomical figure, an engineering figure,an architectural figure or a patent figure, contains certain elementsthat indicate by shape and size the nature of the object the figure isintended to depict. Often, included with these figure are alphanumericreference characters which point to, and are placed next to, the elementfor which the element corresponds. A user viewing the figure typicallyhas to read through a textual description of the figure, which may bemany pages long or in a different location from the figure, to determinewhat element each alphanumeric reference character refers to, in orderto understand the nature of the specific element, as well as the overallfigure. This process may be time-consuming, expensive and error-prone.

While certain aspects of conventional technologies have been discussedto facilitate the present disclosure, no technical aspects aredisclaimed. The claims may encompass one or more of the conventionaltechnical aspects discussed herein.

BRIEF SUMMARY

The present disclosure addresses at least one of the above problems.However, the present disclosure may prove useful in addressing otherproblems and deficiencies in a number of technical areas. Therefore, theclaims, as recited below, should not necessarily be construed as limitedto addressing any of the particular problems or deficiencies discussedherein.

Example embodiments of the present disclosure provide systems, devicesand methods for aiding users in content analysis.

An example embodiment of the present disclosure is acomputer-implemented method which includes identifying a referencewithin a figure and an identifier in a text associated with the figure.The reference referring to an element depicted in the figure. Thereference corresponding to the identifier. The identifier identifyingthe element in the text. The method further includes placing theidentifier on the figure at a distance from the reference. Theidentifier visually associated with the reference upon the placing. Theplacing of the identifier on the figure is irrespective of the distancebetween the identifier and the reference.

In an example embodiment of the present disclosure the identifier isvisually associated with the reference via at least one line displayedon the figure irrespective of the distance between the identifier andthe reference.

In an example embodiment of the present disclosure the at least one lineis colored for visual distinction.

In an example embodiment of the present disclosure the identifier isvisually associated with the reference via a geometric shape displayedon the figure, the shape enclosing the reference and the identifier onthe figure.

In an example embodiment of the present disclosure the shape is coloredfor visual distinction.

In an example embodiment of the present disclosure the identifier iscolored on the figure for visual distinction.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for printing the figureafter the placing of the identifier on the figure, the printed figureincluding both the identifier and the reference.

In an example embodiment of the present disclosure the placing of theidentifier on the figure is user-customizable.

In an example embodiment of the present disclosure the figure and thetext are stored in different locations.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, if the textassociates an another identifier with the reference, placing the anotheridentifier on the figure adjacent to the identifier without overlappingthe identifier.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, receiving thefigure from an image capture device before the identifying of thereference within the figure.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, performing afrequency analysis before the placing of the identifier on the figurewhen the identifier conflicts with an another identifier in the text.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, performing opticalcharacter recognition on the text to aid in identifying the identifier.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, creating abidirectional hyperlink relationship between the reference in the figureand the identifier in the text.

In an example embodiment of the present disclosure the identifier isplaced on the figure on an axis of orientation such that a viewer avoidsrotating the figure to read the identifier.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, translating theidentifier into a language different from the text, the figure includingthe translated identifier.

In an example embodiment of the present disclosure the identifier andthe reference are placed apart from each other in the figure so as tomake readability easier while having a proper scale and being compliantwith at least one of preselected and customized margins.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, avoiding theplacing of the identifier on the figure if the identifier is associatedwith at least one of length, width, depth, volume, diameter, radius,density and direction.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, repeating theprocess for a plurality of references within the figure.

An example embodiment of the present disclosure is acomputer-implemented method which includes identifying a referencewithin a figure and an identifier in a text associated with the figure.The reference referring to an element depicted in the figure. Thereference corresponding to the identifier. The identifier identifyingthe element in the text. The method further includes replacing thereference with the identifier on the figure.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, printing the figureafter the replacing of the reference with the identifier, the printedfigure including the identifier but not the reference.

In an example embodiment of the present disclosure thecomputer-implemented method may further provide for, if the textassociates an another identifier with the reference, placing the anotheridentifier on the figure adjacent to the identifier without overlappingthe identifier.

An example embodiment of the present disclosure is acomputer-implemented method which includes identifying a referencewithin a figure and an identifier in a text associated with the figure.The reference referring to an element depicted in the figure. Thereference corresponding to the identifier. The identifier identifyingthe element in the text. The method further includes placing theidentifier within the element on the figure.

The present disclosure may be embodied in the form illustrated in theaccompanying drawings. Attention is called to the fact, however, thatthe drawings are illustrative. Variations are contemplated as being partof the disclosure, limited only by the scope of the claims. The aboveand other features, aspects and advantages of the present disclosurewill become better understood to one skilled in the art with referenceto the following drawings, detailed description and appended claims.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated into and form a partof the specification, illustrate example embodiments of the presentdisclosure. Together with the detailed description, the drawings serveto explain the principles of the present disclosure. The drawings areonly for the purpose of illustrating example embodiments of the presentdisclosure and are not to be construed as necessarily limiting thedisclosure. Like numbers can refer to like elements throughout. Theabove and other aspects, advantages and features of the presentdisclosure will become better understood to one skilled in the art withregard to the following description, appended claims and accompanyingdrawings where:

FIG. 1 is a flowchart of an example embodiment of a visual associationprocess according to the present disclosure;

FIG. 2 is a flowchart of another example embodiment of a visualassociation process according to the present disclosure;

FIG. 3 is a flowchart of yet another example embodiment of a visualassociation process according to the present disclosure;

FIGS. 4a-4e are diagrams depicting an example embodiment of a process ofvisual association according to the present disclosure;

FIGS. 5a-5c are diagrams depicting another example embodiment of aprocess of visual association according to the present disclosure;

FIGS. 6a-6b are diagrams of an example embodiment of a figure before andafter visual association according to the present disclosure;

FIG. 7 is a network diagram of an example embodiment of a network withinwhich visual association is performed according to the presentdisclosure; and

FIGS. 8a-8b are diagrams of an example embodiment of a figure before andafter visual association according to the present disclosure.

DETAILED DESCRIPTION

The present disclosure will now be described more fully with referenceto the accompanying drawings, in which example embodiments of thedisclosure are shown. The disclosure may, however, be embodied in manydifferent forms and should not be construed as being limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the concept of the disclosure to those skilled in the art.

According to principles of the present disclosure, any verbs as usedherein can imply direct or indirect, full or partial, action orinaction. For example, when an element is referred to as being “on,”“connected” or “coupled” to another element, the element can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present.

Although the terms first, second, etc. may be used herein to describevarious elements, components, regions, layers and/or sections, theseelements, components, regions, layers and/or sections should not belimited by these terms. These terms are only used to distinguish oneelement, component, region, layer or section from another element,component, region, layer or section. Thus, a first element, component,region, layer or section discussed below could be termed a secondelement, component, region, layer or section without departing from theteachings of the present disclosure.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be necessarily limiting of thedisclosure. As used herein, the singular forms “a,” “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises,” “includes” and/or “comprising,” “including” when used inthis specification, specify the presence of stated features, integers,steps, operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure belongs. It willbe further understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Furthermore, relative terms such as “below,” “lower,” “above,” and“upper” may be used herein to describe one element's relationship toanother element as illustrated in the accompanying drawings. It will beunderstood that relative terms are intended to encompass differentorientations of the device in addition to the orientation depicted inthe accompanying drawings. For example, if the device in theaccompanying drawings is turned over, elements described as being on the“lower” side of other elements would then be oriented on “upper” sidesof the other elements. Similarly, if the device in one of the figures isturned over, elements described as “below” or “beneath” other elementswould then be oriented “above” the other elements. Therefore, theexample terms “below” and “lower” can, therefore, encompass both anorientation of above and below.

If any disclosures are incorporated herein by reference and suchincorporated disclosures conflict in part or whole with the presentdisclosure, then to the extent of conflict, and/or broader disclosure,and/or broader definition of terms, the present disclosure controls. Ifsuch incorporated disclosures conflict in part or whole with oneanother, then to the extent of conflict, the later-dated disclosurecontrols.

FIG. 1 is a flowchart of a visual association method according to thefirst embodiment of the present disclosure. A process 100 includesblocks 110-120. Process 100 can be performed via a single core processoror a multi-core processor, irrespective of whether the cores are localto each other.

Block 110 includes matching a reference in a figure to an identifierfound in a text corresponding to the reference which identifies theelement referred to by the reference. The reference in the figure is analphanumeric character visually referring to an element of the figure.One or more alphanumeric characters may be used, or evennon-alphanumeric character references may be used. The reference can beor include symbols as well. An identifier is a name or brief descriptionof the element, which is often textually described. Alternatively, theidentifier can even include a plurality of terms, a sentence or even aparagraph.

Typically, the name of the element is disclosed in a description of thefigure. For example, in a figure of a patent application, the number 10,which is a reference, can visually refer to an element of the figure,such as a chair. The word “chair” is an identifier that is disclosed inthe specification describing the figure of the patent application.

Block 120 includes visually associating in the figure the identifierwith the reference. Block 120 can be performed close in time or farapart in time to block 110. One way of visually associating theidentifier with the reference is by placing the identifier adjacent tothe reference. Alternatively, non-adjacent visual association ispossible as well where the identifier refers to the referenceirrespective of where the identifier is placed on the figure. Thus, theterm “chair” does not have to be adjacent to reference 10. As long asthere is a visual association between the term “chair” and reference 10,even if the term “chair” is at a far distance from reference 10, suchas, for example, at a corner of the page of the figure, a bottom centeror top center of the page of the figure, along a left or right side ofthe page of the figure, a user can easily identify what the reference 10is referring to. An example of an adjacent visual association is if thenumber 10 in the figure refers to a chair, then the word “chair” isplaced adjacent to the number 10. hus, a viewer of the figure, such as astudent, a scientist, a hobbyist, an engineer or a patent professional,can easily identify what the reference 10 is referring to without havingto peruse the text to find the identifier. Visual associating anidentifier with a corresponding reference, even when the two are notadjacent, is described herein.

FIG. 2 is a flowchart of a visual association method according to thefirst embodiment of the present disclosure. A process 200 includesblocks 210-250.

Block 210 includes searching within a figure for a reference referringto an element of the figure. One way of performing the searching is viacomputer vision or computer pattern recognition, such as opticalcharacter recognition (OCR). The computer searches the figure to locatereferences, such as alphanumeric and non-alphanumeric charactersreferring to elements in the figure. In an example embodiment, thesearching within the figure for the reference can be within selectivegeographic regions of the figure. A determination can be made of whichselective geographic regions of the figure can be performedautomatically, via a preset rule or manually. Alternatively, the figurecan be searched via text searching.

Block 220 includes copying the found references into a data structure.One example of a data structure is a list or an array. One example ofthe found reference is an alphanumeric character, such as the numbers 10or 20. Block 230 includes searching within a text describing the figurefor an identifier corresponding to the reference. Although typically asingle reference refers to a single identifier, a possibility exists ofanother type of correspondence, such as one to many, many to one, ormany to many. In such a case, either an error message is generated anddisplayed, for example, adjacent to the reference. Alternatively, a modecomputation/frequency analysis with respect to the reference or theidentifier is made, from which it is determined which identifier shouldbe displayed adjacent to a reference, the mode term is flagged and themode term is used for any subsequent blocks. The flagging can be usedlater to visually indicate potential imprecision of visual association.

In an example embodiment of the present disclosure, the searching withinthe text describing the figure can be done within only selectiveportions of the description as selected by the user, whether a human ora machine/software. A determination of which selective portions of thedescription can be made automatically via a preset rule or manually.

Block 240 includes copying the found identifier into the data structure.Block 250 includes visually associating in the figure the identifierwith the reference.

FIG. 3 is a flowchart of yet another example embodiment of a visualassociation process according to the present disclosure. A process 300includes blocks 310-340.

Block 310 includes searching within a text describing a figure for anidentifier corresponding to a reference referring to an element withinthe figure. The text searching can include OCR.

Block 320 includes storing the found identifier in a data structure.

Block 330 includes searching the figure for the reference. One way ofsearching the figure for the reference is to create a map of locationswhere in the figure the references are located. Another way of searchingthe figure for the reference is to search for a presence of just thereference.

Block 340 includes visually associating in the figure the storedidentifier with the found reference.

FIGS. 4a-4e are diagrams depicting an example embodiment of a process ofvisual association according to the present disclosure.

FIG. 4a depicts a patent figure prior to visual association. Althoughdepicted figure is a patent figure, other types of figures, such asarchitectural, engineering, anatomical, scientific, historical,blueprints, financial or geographical figures, having a textualdescription of the figures can be used as well. Any type of content canbe depicted in the figure. The figures can be any types of diagrams,flowcharts or tree diagrams. The figures can be depicted in any type ofviews, such as a side view, a perspective view, a top view or bottomviews. The figures can be grayscale, white/black or color. The figurescan be linked or broken into a plurality of sub-figures depicting oneobject together. The figures can be drawn by hand, created via acomputer or automatically drawn.

FIG. 4b depicts references stored within a data structure, such as atable or an array. The references are obtained from analyzing, via acomputer, FIG. 32 as depicted in FIG. 4a . The analyzing can beperformed via OCR or other processes as known in the art.

FIG. 4c depicts descriptive text, such as a patent detailed description,describing elements depicted in FIG. 4a . The elements are referenced bythe references shown in FIG. 4a and stored within the data structure ofFIG. 4b . The descriptive text can be stored in the same file as FIG. 32as depicted in FIG. 4a or the descriptive text can be stored in a filedifferent file, whether one a same computer or a different computer,from the file containing FIG. 32 as depicted in FIG. 4a .

FIG. 4d depicts the data structure after the descriptive text depictedin FIG. 4c has been parsed and matched accordingly, which can occur inone or more steps/processes. As shown in FIG. 4d , paragraph [0079] hasbeen parsed according to the references stored in the data structure andcorresponding identifiers are stored in the data structure. Thus, thedata structure stores the identifiers corresponding to the references.

FIG. 4e depicts different ways of visually associating the identifierswith the references.

Identifier “first database” is placed adjacent to reference 10 using aline. The identifier can be in the same font or font size as the rest ofthe figure, as determined automatically via a computer or manually via auser, or the font or font size can be different, as determinedautomatically via a computer or manually via a user. The identifier, thereference or the line can be highlighted. The line can also visuallyassociate a plurality of elements. The line can be a straight line or acurved/zigzag line. The line can be without any gaps or the line can bedefined via a plurality of closely spaced elements, which can be iconic,symbolic or alphanumeric. The line can be a plurality of aligned orparallel lines. The line can be placed over other elements or avoidplacement over other elements, references or identifiers. The computercan be programmed to determine how to properly place the line, such asto place or avoid placing over other elements, references oridentifiers. Alternatively, the user can select how to properly placethe line or maneuver/drag the line on the figure. The line, thereference or the identifier or any of their portions can be of anycolor. A user or a computer can automatically select colors. The linecan be colored to be visually distinct from the reference, theidentifier or the element or other elements, references or identifiers.The line can be hyperlinked, whether uni-directionally orbi-directionally. Upon clicking, the hyperlink can lead to otherelements, references and identifiers whether in the present figure,other figures, the text description or other text descriptions. Uponclicking, the hyperlink can allow for popups, hover-overs or slide-outsto disclose information relating to the element, reference or identifieror other elements, references or identifiers.

In an alternative example embodiment, visual association can beperformed via placing the identifier adjacent to the reference andplacing a shape, such as a rectangle, a box, a circle, an oval, atrapezoid or any other shape, over the reference and the identifier onthe figure. The shape can fully or partially encompass the identifierand the reference. The shape delineation, the shape background orforeground, the identifier, the reference or any of their portions canbe colored for visual distinction. The shape can be defined via a singleline or a plurality of lines, dots, minuses, pluses or other visualelements, including alphanumeric characters. The shape can be a bubble,whether a popup, a hover-over or slide-out. The shape can behyperlinked. The identifier can be hyperlinked, whetheruni-directionally or bi-directionally. Upon clicking, the hyperlink canlead to other elements, references and identifiers whether in thepresent figure, other figures, the text description or other textdescriptions. Upon clicking, the hyperlink can allow for popups,hover-overs or slide-outs to disclose information relating to theelement, reference or identifier or other elements, references oridentifiers.

Identifier “CPU” replaces the reference 20 as depicted in FIG. 4a . Theidentifier can be in the same font or font size as the rest of thefigure, as determined via automatically via a computer or manually via auser, or the font or font size can be different, as determined viaautomatically via a computer or manually via a user. The identifier canalso visually associate a plurality of elements. The identifier, thereference or the line can be highlighted. The identifier can be placedover other elements or avoid placement over other elements, referencesor identifiers. The computer can be programmed to determine how toproperly place the line, such as to place or avoid placing over otherelements, references or identifiers. Alternatively, the user can selecthow the identifier replaces the reference in the figure. The identifieror any of its portions can be of any color. A user or a computer canautomatically select colors. The identifier can be colored to bevisually distinct from the reference, the identifier or the element orother elements, references or identifiers. The identifier can behyperlinked, whether uni-directionally or bi-directionally. Uponclicking, the hyperlink can lead to other elements, references andidentifiers whether in the present figure, other figures, the textdescription or other text descriptions. Upon clicking, the hyperlink canallow for popups, hover-overs or slide-outs to disclose informationrelating to the element, reference or identifier or other elements,references or identifiers.

Identifier “second database” is placed within the element correspondingto the reference 30. The element, such as its location, size or shape,is automatically determined by a computer using various softwarealgorithms as known in the art. These algorithms can employ computervision/pattern recognition. The algorithms can refer to element libraryas publically or privately available. Such library can be stored on thecomputer or available via the Internet. The algorithms can alsodetermine the element via determining meaning of the identifier aslooked up in internal or external library/database. The element can befilled with color for visual distinction. The color can be manuallyselected by a user or the color can be automatically selected by acomputer. A plurality of identifiers, whether identifying same ordifferent element, can be placed within the element and can be visuallydistinct from other elements, references and identifiers. Theidentifier, the reference or the line can be highlighted. The identifiercan be in the same font or font size as the rest of the figure, asdetermined via automatically via a computer or manually via a user, orthe font or font size can be different, as determined via automaticallyvia a computer or manually via a user. The identifier can also visuallyassociate a plurality of elements. The identifier can be placed overother elements or avoid placement over other elements, references oridentifiers. The computer can be programmed to determine how to properlyplace the line, such as to place or avoid placing over other elements,references or identifiers. Alternatively, the user can select how theidentifier replaces the reference in the figure. The identifier, thereference or any of their portions can be of any color. The identifiercan be colored to be visually distinct from the reference, theidentifier or the element or other elements, references or identifiers.The identifier can be hyperlinked, whether uni-directionally orbi-directionally. Upon clicking, the hyperlink can lead to otherelements, references and identifiers whether in the present figure,other figures, the text description or other text descriptions. Uponclicking, the hyperlink can allow for popups, hover-overs or slide-outsto disclose information relating to the element, reference or identifieror other elements, references or identifiers.

Regardless of visual association, a user can select or a computer canautomatically decide to shrink or compact the figure so as to allow forplacement of the identifier or a plurality of identifier so as to allowfor readability of the identifier or the plurality of the identifiers.For example, font sizes can be automatically increased.

Any method of visual association can allow for any portion of anyelement, identifier, reference, line, shape, character, symbol, tag,hyperlink or any other way of visual association to be of any color orany color for visual distinction. Any of these types of visualassociation can be automatically or manually combined in any way and anyof these types of visual association can be automatically or manually bemade visually distinct from other references or identifiers. Forexample, a computer can automatically determine how to visuallyassociate and such determination can mix and match different types ofvisual associations. Such mix and match can depend on the context orcontent of the figure, such as whether to write over or avoid writingover other elements, references or identifiers. One element can bevisually associated with all or less than all ways of visuallyassociating.

FIGS. 5a-5c are diagrams depicting another example embodiment of aprocess of visual association according to the present disclosure.

FIG. 5a depicts descriptive text, such as a patent detailed description,describing various elements in a corresponding figure, in which theelements are referenced by references and named via identifiers.

FIG. 5b depicts a data structure after the descriptive text depicted inFIG. 5a has been parsed, matched and stored in the data structure. Asshown in FIG. 5b , paragraph [0079] has been parsed and matched by thereferences and corresponding identifiers and stored in the datastructure. Thus, the data structure stores the identifiers correspondingto the references.

FIG. 5c depicts different ways of visually associating the identifierswith the references. Identifier “first database” is adjacent toreference 10. Identifier “CPU” replaces the reference 20. Identifier“second database” is placed within the element corresponding to thereference 30.

Any of these types of visual association can be automatically ormanually combined in any way, even with FIGS. 4a -4 e, and any of thesetypes of visual association can be automatically or manually be madevisually distinct from other references or identifiers.

FIGS. 6a-6b are diagrams of an example embodiment of a figure before andafter visual association according to the present disclosure. FIG. 6adepicts a microphone pen before visual association. FIG. 6b depicts themicrophone pen after visual association. Each identifier as depicted inFIG. 6b can be visually associated with a reference as depicted in FIG.6a that the identifier replaced. For example, as shown in FIG. 6b , theidentifier “chamber” can be visually associated with the reference 204using any visual association methods as described herein.

FIG. 7 is a network diagram of an example embodiment of a network withinwhich visual association is performed according to the presentdisclosure. A network 700 includes a user computer 710 connected to anetwork, such as the Internet. A first server 720 and a second server730 are accessible via the network.

Any one or all or computer 710 and servers 720 can be any type of acomputer, such as a desktop, a laptop, a mainframe, a cloud-computingsystem, a smartphone, a tablet computer or a workstation.

Visual association, as described herein, can be performed locally onuser computer 710 by a program installed on a hard disk or can be run asa module within other software, such as a word processing application, abrowser or a mobile app. Alternatively, visual association can beperformed via a website or a web portal. Alternatively, visualassociation can be performed by first server 720 wherein a user of usercomputer 710 accesses first server 720, performs the visual associationon a selected figure or a file and then downloads the visuallyassociated figure or the file. More alternatively, visual associationcan be performed by first server 720 on a set files which are thenstored in a database on second server 730. Then, a user of user computer710 accesses second server 730 to download a selected visuallyassociated figure or a visually associated file.

The visual associating may include printing the visually associated fileor a figure of the file or a section of the figure. When printingmultiple pages with the visually associated figures on the same sheet,the visual association of one page avoids interfering with visualassociation of other sheets. Also, the visually associating can beperformed according to a view of the figures, wherein the view is aportrait view or a landscape view.

In an example embodiment, the visually associating is performedaccording to a preset rule, such as placing the identifier a certaindistance from the reference or visually associating in a way such thatall identifiers fit on a single screen or a page. The distance can bemanually selected. Alternatively, the distance can be automaticallyselected by a computer upon analysis of a section of the figure or thefigure to determine optimal placement and/or method of visualassociation.

In an example embodiment, in a computer network environment, one usercan perform a visual association process, store and allow access to thevisually associated file (old one or create new one) to other users.Thus, other users can avoid repetition of the visual association processin order to improve efficiency.

In an example embodiment, upon matching of the references andidentifiers, the method can further display or be used for data mining,such as determining which elements have missing references oridentifiers.

In an example embodiment, the visual associating can be performed on asection of a figure, a single FIGURE, multiple figures within a file, asingle FIGURE within each of multiple files, multiple figures inmultiple files, or via a directory within memory storing a plurality offiles with figures.

Even though the figure and the description can be stored in one computerfile, the figure and the description can be stored in multiple computerfiles, in one or multiple computers, and/or in one or multiple distinctlocales. Further, the figure can be obtained from an image capturedevice, such as a scanner, and matched with the description. Likewise,the description can be automatically or manually obtained from adescription database, such as a patent text database or a digitaltextbook, and then automatically or manually matched with the figure.Also, although a single FIGURE and a single description are described,multiple figures can be one described as one figure and one figure canbe described in multiple descriptions and if identifiers conflict, thena frequency analysis can be used or a preset rule can be invoked.Moreover, if a description of the figure is an image, then text withinthe image can be recognized via OCR technology and then parsed asdescribed herein.

In an example embodiment, in the figure, by selecting the reference, theelement or a portion of the element itself, such as by clicking on theelement/reference or by hovering the mouse over the element/reference,the user may cause to be dynamically displayed, such as adjacent to thereference or element or visually associated with the reference/element,an identifier associated with a respective reference. Since each elementor a portion thereof is associated with a different reference, movingthe mouse from one element to another enables the reference associatedwith the another element to be displayed as described herein.

In an example embodiment, in a patent application or a patent grantstored in a computer accessible file, if the user selects at least oneword in a claim and that selected word is shown in a figure, asdetermined by the parsing and identification from the description andlocation via the reference in the figure, then the element in the figureand/or the numeral corresponding to the element will be highlighted orshown in a new window or bubble or any other type of data display thatinform the user of the location of the at least one word in the claim.This allows the user to quickly find the associated figure or element inthe figure for further understanding of the text. Similarly, the usercan do the reverse, whereby the user selects an element of the figure,which highlights a corresponding associated text in the description orthe concept in a claim, such as a noun, by showing in a new window orbubble or any other type of data display that inform the user of thelocation of the at identifier.

In an example embodiment, in a patent application or a patent grantstored in a computer accessible file, after parsing and visuallyassociating, the data can be viewed via calendar views, such as for acontinuation-in-part patent application where a date or dates of filingor priority can be associated with references/identifiers to identifynewly added subject matter, alerts, such as via conflictingreferences/identifiers, information bubbles associated withreferences/identifiers, color variances for references/identifiers, suchas user-customizable color palettes for each or all or any as desiredreferences/identifiers.

In an example embodiment, in a figure, after parsing and correspondingreferences to identifiers or vice versa, a listing of references andcorresponding identifiers can be displayed on a side of the figure orcorner of the page or anywhere else away from the actual figure in aform of a table or any other representation of data that allows the userto easily identify which identifiers the references refer to. This canbe done simultaneously with or alternatively to the visual associationas described herein.

In an example embodiment, a figure or a plurality of figures can beadjusted to have a same or similar axis of orientation to allow forconvenient reading of the figure. Likewise, in one or more figures,references or identifiers can be placed or adjusted or rotated or movedto have a similar axis of orientation so as to be viewed withoutrotating the figure or the figures. Fonts or font sizes can beautomatically adjusted as well.

In an example embodiment, after parsing and matching the identifiers andthe references on at least one figure, the user can click thereference/identifier to jump or lead to a first instance of suchreference/identifier in the description or the claims or the figure.

In an example embodiment, after parsing and matching the identifiers andthe references on at least one figure, upon clicking/hoveringover/selecting the reference, a scrollable/expandable/window with adescription of at least a portion of the figure or a specificdescription of the element corresponding to the selected reference isshown.

In an example embodiment, whether before, during or after parsing andmatching the identifiers and the references on at least one figure or asection of the figure, at least one of the references or identifiers inthe figure can be translated into any human language. The language canbe as selected from a menu provided to a user or automatically detectedvia a computer, whether local or remote, and then facilitatetranslation. The translation can occur via using online translationengine, such as Google Translate, or locally, via locally storedtranslation library or using a computer's operating system. Thetranslation can occur before, during or after the placing of theidentifier on the figure.

In an example embodiment, in a patent application or a patent grantstored in a computer accessible file, manually select a plurality ofwords from at least one independent claim or automatically parse atleast one independent claim into a plurality of nouns and match viasearching the plurality of words or the parsed nouns to at least onevisually associated figure that contains the plurality of words or theparsed nouns in another patent application or another patent grantstored in another computer accessible file on a network source, such asa database hosted on a server. Also, any other visually associatedfigures in other literature or websites or applications can be matchedto as well. Thus, this method can be used to identify an anticipationrejection compliant with at least US patent law.

In an example embodiment, in a patent application or a patent grantstored in a computer accessible file, manually select a plurality ofwords from at least one independent claim or automatically parse atleast one independent claim into a plurality of nouns and match viasearching the plurality of words or the parsed nouns to at least onefigure in the same file. Then, at least temporarily hiding, such as viaplacing white space or color contrasting or putting an X through, thereferences or the identifiers in the figure that are not associated orcorrespond to the nouns or the words. Thus, only references havingidentifiers associated with the claim are shown in the figure.

In an example embodiment, references and identifiers are placed apartfrom each other in the figure so as to make readability easier whilebeing compliant with preselected or customized margins and having aproper scale.

In an example embodiment, some measurements, such as length, width,depth, volume, diameter, radius, density, direction, can remainunlabeled. Such measurements can be detected by presence of varioussigns, such as arrows on the figure or when the text identifies theidentifiers as such.

In an example embodiment, via the disclosure the user can determinewhether a claimed element is in the specification or the figure.

In an example embodiment, an examiner can put on 3 d glasses, such asmade by Nvidia, and perform any disclosures provided herein withoutrunning visual association process on the actual file having referencesand identifiers. Rather, the disclosure as described herein is performedby the software for the glasses.

In an example embodiment, the disclosed technology can ensure a figure'scompliance with 37 CFR 1.83 or 1.84 and display warnings if the figureis not compliant. For example, if the figure has improper margins,fonts, font sizes, colors, the disclosed technology can notifynon-compliance with 37 CFR 1.83 or 1.84.

In an example embodiment, the disclosure can be performed on one file, aplurality of files or portions retrieved from a plurality of files.Also, the disclosure can be performed via one or a plurality ofcomputers or servers. Also, the files can be stores on one computer or aplurality of computers in any way. Also, the disclosure can be performedlocally or remotely or on one computer or a software app or over acomputer network, such as the Internet.

In an example embodiment, visual association can be performed on a videoshowing a plurality of images or figures where the video is associatedwith text mentioning the elements as shown in the video. The video canhave audio reading the text.

In an example embodiment, any portion of any embodiments or permutationsthereof, as described herein, can be combined in any way according tothe principles of the present disclosure.

FIGS. 8a-8b are diagrams of an example embodiment of visual associationa figure before and after visual association according to the presentdisclosure. Any of the methods of visual association can be combined inany way. For example, although one figure can be visually associated inone method of visual association, the one figure can include multiplemethods of visual association. When a plurality of figures is desired tobe visually associated, then all or less than all figures can beassociated in a same or different ways. Any elements, references,identifiers, methods of visual associations or portions thereof can behyperlinked. When a computer decides which visual method to employ, thenthe computer uses algorithms which look for presence of empty space,such as white space, near the reference to place the identifier,possibility of reference/visual association placement over otherelements, references, identifiers or methods of visual association, sizeof the figure or portion of the figure, screen space, font size, colors,speed/memory of computer or web connection, user confusion (as definedby a user or programmed in advance) and other similar concerns.

Note that reference 180 within the element has been replaced withidentifier “base” within the element corresponding to reference 180. Theidentifier can be colored for visual distinction or be same color as atleast a portion of the figure. A user can select or a computer canautomatically determine as to how to most optimally visually associate.With the reference 180, the computer automatically chose to write withinthe element.

Note that references 96 and 120 outside of their elements have beenreplaced with identifiers “cylinder” and “center” outside of theirelements corresponding to references 96 and 120. Alternatively, suchreplacement could be done within the elements, like done regardingreference 180. The identifiers can be colored for visual distinction orbe same color as at least a portion of the figure. A user can select ora computer can automatically determine as to how to most optimallyvisually associate. With the references 96 and 120, the computerautomatically chose to replace the reference.

Note that reference 206 has been visually associated via an alphanumericcharacter corresponding to a plus symbol. Alternatively,non-alphanumeric character, such as a symbol or an icon, can also beused. The character or the reference can be can be colored for visualdistinction or be same color as at least a portion of the figure. A usercan select or a computer can automatically determine as to how to mostoptimally visually associate. With the reference 206, the computerautomatically chose to write within the element.

Note that reference 140 has been visually associated via a line definedvia a broken line visually associating over at least one element. Theline indicates the identifier “drum” corresponding to the reference 140.The line, the reference or identifier can be can be colored for visualdistinction or be same color as at least a portion of the figure. A usercan select or a computer can automatically determine as to how to mostoptimally visually associate. With the reference 140, the computerautomatically chose to use a broken line method over the at least oneelement.

Note that reference 128 has been visually associated via an identifier“rod” placed near the reference 128 and a shape placed around thereference 128 and the identifier “rod.” The shape can be a rectangle, abox, a circle, an oval, a trapezoid or any other shape. The shape canfully or partially encompass the identifier and the reference. The shapedelineation, the shape background or foreground, the identifier or thereference can be colored for visual distinction. The shape can bedefined via a single line or a plurality of lines, dots, minuses, plusesor other visual elements, including alphanumeric characters. The shapecan be a bubble, whether a popup, a hover-over or slide-out. The shapecan be hyperlinked. The identifier can be hyperlinked, whetheruni-directionally or bi-directionally. Upon clicking, the hyperlink canlead to other elements, references and identifiers whether in thepresent figure, other figures, the text description or other textdescriptions. Upon clicking, the hyperlink can allow for popups,hover-overs or slide-outs to disclose information relating to theelement, reference or identifier or other elements, references oridentifiers. A user can select or a computer can automatically determineas to how to most optimally visually associate. With the reference 128,the computer automatically chose to the shaping method.

Note that reference 126 has been visually associated with identifier“pipe” via an unbroken line in a non-adjacent manner i.e. irrespectiveof the distance between the reference 126 and the identifier “pipe.” Theelement, the line, the reference, the identifier or any portions thereofcan be colored for visual distinction or be same color as at least aportion of the figure. A user can select or a computer can automaticallydetermine as to how to most optimally visually associate. With thereference 126, the computer automatically chose to the unbroken linemethod.

Note that reference 166 has been visually associated with identifier“plate” via a broken line in a non-adjacent manner i.e. irrespective ofthe distance between the reference 126 and the identifier “pipe.” Theelement, the line, the reference, the identifier or any portions thereofcan be colored for visual distinction or be same color as at least aportion of the figure. A user can select or a computer can automaticallydetermine as to how to most optimally visually associate. With thereference 166, the computer automatically chose to the broken linemethod to associate over other elements and association over otherreferences, identifiers, methods of visual association or any portionsthereof can also be performed.

As will be appreciated by one skilled in the art, aspects of the presentdisclosure may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present disclosure may take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present disclosure may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate or transport a program for use by or in connection with aninstruction execution system, apparatus or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. Other types of programming languages includeHTML5, Flash and other similar languages. The program code may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Aspects of the present disclosure are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality and operation of possible implementations ofsystems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present disclosure has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the disclosure in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the disclosure. Theembodiment was chosen and described in order to best explain theprinciples of the disclosure and the practical application, and toenable others of ordinary skill in the art to understand the disclosurefor various embodiments with various modifications as are suited to theparticular use contemplated.

The flow diagrams depicted herein are just one example. There may bemany variations to this diagram or the steps (or operations) describedtherein without departing from the spirit of the disclosure. Forinstance, the steps may be performed in a differing order or steps maybe added, deleted or modified. All of these variations are considered apart of the claimed disclosure.

While the preferred embodiment to the disclosure had been described, itwill be understood that those skilled in the art, both now and in thefuture, may make various improvements and enhancements which fall withinthe scope of the claims which follow. These claims should be construedto maintain the proper protection for the disclosure first described.

The description of the present disclosure has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications and variations in techniques and structures will beapparent to those of ordinary skill in the art without departing fromthe scope and spirit of the disclosure as set forth in the claims thatfollow. Accordingly, such modifications and variations are contemplatedas being a part of the present disclosure. The scope of the presentdisclosure is defined by the claims, which includes known equivalentsand unforeseeable equivalents at the time of filing of this application.

What is claimed is:
 1. A system comprising: a processor programmed to:cause a retrieval of a patent document including a patent figure and apatent text describing the patent figure, wherein the patent figureshows a first part number in a first color, wherein the patent textincludes a second part number and a part name associated with the secondpart number, wherein the first part number is associated with the partname based on the first part number matching the second part number; andcause a presentation of (a) a copy of the part name over the patentfigure in a second color and (b) a line extending over the patent figurebetween the first part number and the copy based on the retrieval. 2.The system of claim 1, wherein the line is presented over the patentfigure in the second color.
 3. The system of claim 1, wherein theprocessor is programmed to: cause a presentation of a listing includinga set of part numbers and a set of part names lateral to the first partnumber shown in the first color in the patent figure and the copy of thepart name presented over the patent figure in the second color, whereinthe set of part numbers includes a copy of the first part number or thesecond part number, wherein the set of part names includes the copy ofthe part name, wherein the listing presents the set of part numbersvisually corresponding to the set of part names.
 4. The system of claim3, wherein the copy of the part name presented over the patent figure inthe second color is highlighted based on the copy of the part name inthe set of part names or the copy of the first part number or the secondpart number in the set of part numbers being user-selected.
 5. Thesystem of claim 1, wherein the patent text is recited in a first humanlanguage, wherein the processor is programmed to present the copy of thepart name as translated from the first human language to a second humanlanguage over the patent figure in the second color.
 6. The system ofclaim 1, wherein the first part number is recognized in the patentfigure before the second part number is identified in the patent textsuch that the second part number is identified based on matching to thefirst part number.
 7. The system of claim 1, wherein the first partnumber is recognized in the patent figure after the second part numberis identified in the patent text such that the first part number isrecognized based on matching to the second part number.
 8. A systemcomprising: a processor programmed to: cause a retrieval of a patentdocument including a patent figure and a patent text describing thepatent figure, wherein the patent figure shows a first part number,wherein the patent text includes a second part number and a part nameassociated with the second part number, wherein the first part number isassociated with the part name based on the first part number matchingthe second part number; and cause a presentation of (a) a copy of thepart name over the patent figure and (b) a listing including a set ofpart numbers and a set of part names lateral to the first part numbershown in the patent figure and the copy of the part name presented overthe patent figure based on the retrieval, wherein the set of partnumbers includes a copy of the first part number or the second partnumber, wherein the set of part names includes the copy of the partname, wherein the listing presents the set of part numbers visuallycorresponding to the set of part names, wherein the copy of the partname presented over the patent figure is highlighted based on the copyof the part name in the set of part names or the copy of the first partnumber or the second part number in the set of part numbers beinguser-selected.
 9. The system of claim 8, wherein the copy of the partname presented over the patent figure lateral to the listing is coloreddifferently from the first part number in the figure lateral to thelisting.
 10. The system of claim 8, wherein the processor is programmedto cause a presentation of a line extending over the patent figurebetween the first part number and the copy of the part name lateral tothe listing.
 11. The system of claim 10, wherein the line is coloreddifferently from the first part number in the figure lateral to thelisting.
 12. The system of claim 8, wherein the patent text is recitedin a first human language, wherein the processor is programmed topresent the copy of the part name as translated from the first humanlanguage to a second human language over the patent figure lateral tothe listing.
 13. The system of claim 8, wherein the patent text isrecited in a first human language, wherein the processor is programmedto present the copy of the part name as translated from the first humanlanguage to a second human language in the set of part names lateral tothe first part number shown in the patent figure and the copy of thepart name presented over the patent figure.
 14. The system of claim 8,wherein the first part number is recognized in the patent figure beforethe second part number is identified in the patent text such that thesecond part number is identified based on matching to the first partnumber.
 15. The system of claim 8, wherein the first part number isrecognized in the patent figure after the second part number isidentified in the patent text such that the first part number isrecognized based on matching to the second part number.
 16. A systemcomprising: a processor programmed to: cause a retrieval of a patentdocument including a patent figure and a patent text describing thepatent figure in a first human language, wherein the patent figure showsa first part number, wherein the patent text includes a second partnumber and a part name associated with the second part number, whereinthe first part number is associated with the part name based on thefirst part number matching the second part number; and cause apresentation of the part name as translated from the first humanlanguage to a second human language over the patent figure based on theretrieval.
 17. The system of 16, wherein the part name is presented inthe second human language over the patent figure while being coloreddifferently from the first part number.
 18. The system of 16, whereinthe processor is programmed to present a line over the patent figureextending between the first part number and the part name presented inthe second human language over the patent figure.
 19. The system of 18,wherein the line is colored differently from the first part number. 20.The system of 16, wherein the processor is programmed to: cause apresentation of a listing including a set of part numbers and a set ofpart names recited in the second human language lateral to the firstpart number and the part name presented over the patent figure in thesecond human language, wherein the set of part numbers includes a copyof the first part number or the second part number, wherein the set ofpart names include a copy of the part name as translated from the firsthuman language to the second human language, wherein the listingpresents the set of part numbers visually corresponding to the set ofpart names in the second human language.